Skip to content

feat: allow easy inversion of tests#82

Open
halvko wants to merge 2 commits intomasterfrom
feat/allow-easy-inversion-of-tests
Open

feat: allow easy inversion of tests#82
halvko wants to merge 2 commits intomasterfrom
feat/allow-easy-inversion-of-tests

Conversation

@halvko
Copy link
Copy Markdown
Contributor

@halvko halvko commented Apr 30, 2026

I would like to easily be able to invert a test. Look at the tests for what this actually does, or if you want a very AI generated description:

Feature request: fails support in testActor

The use case

Sometimes a scraper relies on a platform-side behaviour that can silently stop working (e.g. sending an undocumented API intent token that Facebook used to honour but no longer does). When that happens, a test that was asserting correct behaviour starts failing. The desired response is:

  1. Acknowledge the regression — don't just delete or skip the test.
  2. Keep a sentinel in CI that will alert you if the behaviour resumes, so you can re-enable the full assertion.

The current workaround is to manually invert every assertion with .not, add a prose comment explaining the inversion, and rename the test with a warning in the title. That's fragile and easy to misread.

What Vitest already provides

Vitest has test.fails(). A test wrapped with it is expected to throw/fail. If it does fail, the suite reports it as passed. If it unexpectedly passes, the suite reports it as failed — exactly the sentinel behaviour we want. The symmetry with test.skip() / test.todo() makes the intent immediately legible to any Vitest user.

Why it can't be used today

testActor is the only public entry point for platform tests, and internally it calls:

vitestTest.runIf(shouldRun)(name, options, async (context) => { … });

vitestTest.runIf(condition) returns a plain callable; you can't chain .fails() onto it. So consumers have no way to reach vitestTest.fails.runIf(shouldRun)(…) without bypassing testActor entirely — which means losing the build-pinning logic, the run() helper, and the annotate calls.

Proposed change

Add an optional fails flag to ActorTestOptions (which already extends Vitest's TestOptions, so the pattern is familiar):

export type ActorTestOptions = Omit<TestOptions, 'retry'> & {
    retry?: TestOptions['retry'];
    /**
     * Mark this test as expected to fail (wraps the underlying vitest test with `test.fails`).
     * The test passes as long as it keeps failing, and alerts you (by failing) if it starts passing.
     * Use this to keep a sentinel for known regressions without inverting assertions by hand.
     */
    fails?: boolean;
};

Then in testActor in lib.ts, swap the test function based on the flag:

const vitestTestFn = options.fails
    ? vitestTest.fails.runIf(shouldRun)
    : vitestTest.runIf(shouldRun);

vitestTestFn(name, options, async (context) => { … });

vitestTest.fails is itself a full TestAPI object (same shape as vitestTest), so .runIf() is available on it and the rest of the body is unchanged.

What the call-site then looks like

testActor(
    ACTOR_ID,
    'Newest sorting works for posts that do not expose it in the UI',
    async ({ expect, run }) => {
        // … run actor …
        expect(results[0]?.profileName).toBe('Leif Andersson');
        expect(results[10]?.profileName).toBe('Pål Kvitberg');
    },
    { fails: true }, // Facebook stopped honouring REVERSE_CHRONOLOGICAL_UNFILTERED_INTENT_V1
                     // for posts that don't advertise it. Remove `fails` when it works again.
);

The assertions stay in their natural (positive) form, the inversion is expressed as a single structured option rather than spread across comments and .not calls, and any Vitest user reading the test immediately understands the intent.

@halvko halvko marked this pull request as ready for review April 30, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants